Counterfactual Reasoning and Probabilistic Methods for Trustworthy AI

Applications in Finance

Patrick Altmeyer

The Problem with Today’s AI

From human to data-driven decision-making …

  • Black-box models like deep neural networks are being deployed virtually everywhere.
  • Includes safety-critical and public domains: health care, autonomous driving, finance, …
  • More likely than not that your loan or employment application is handled by an algorithm.

… where black boxes are recipe for disaster.

  • We have no idea what exactly we’re cooking up …
    • Have you received an automated rejection email? Why didn’t you “mEet tHe sHoRtLisTiNg cRiTeRia”? 🙃
  • … but we do know that some of it is junk.
Figure 1: Adversarial attacks on deep neural networks. Source: Goodfellow, Shlens, and Szegedy (2015)

Towards Trustworthy AI

  • Machine Learning is increasingly being utilized in Finance and Economics.
  • While this promises to generate many beneftis, it also entails great risks and challenges for regulators, market participants and society at large.
  • In this PhD project we plan to identify and tackle some of these challenges through methodological contributions and applied work.
  • We focus in particular on Probabilistic Models and Counterfactual Reasoning.

Ground Truthing

Probabilistic Models

Counterfactual Reasoning

Supervisors

Figure 2: Cynthia C. S. Liem (promotor and daily supervisor)
Figure 3: Arie van Deursen (promotor)

Counterfactual Explanations

Explaining Black-Box Models through Counterfactuals

Stable Dev Build Status Coverage

CounterfactualExplanations.jl is a package for generating Counterfactual Explanations (CE) and Algorithmic Recourse (AR) for black-box algorithms. Both CE and AR are related tools for explainable artificial intelligence (XAI). While the package is written purely in Julia, it can be used to explain machine learning algorithms developed and trained in other popular programming languages like Python and R. See below for short introduction and other resources or dive straight into the docs.

Turning a nine (9) into a four (4).

A sad 🐱 on its counterfactual path to its cool dog friends.

Endogenous Macrodynamics in AR - motivation

TL;DR: We find that standard implementation of various SOTA approaches to AR can induce substantial domain and model shifts. We argue that these dynamics indicate that individual recourse generates hidden external costs and provide mitigation strategies.

In this work we investigate what happens if Algorithmic Recourse is actually implemented by a large number of individuals.

Figure 4 illustrates what we mean by Endogenous Macrodynamics in Algorithmic Recourse:

  • Panel (a): we have a simple linear classifier trained for binary classification where samples from the negative class (y=0) are marked in blue and samples of the positive class (y=1) are marked in orange
  • Panel (b): the implementation of AR for a random subset of individuals leads to a noticable domain shift
  • Panel (c): as the classifier is retrained we observe a corresponding model shift (Upadhyay, Joshi, and Lakkaraju 2021)
  • Panel (d): as this process is repeated, the decision boundary moves away from the target class.
Figure 4: Proof of concept: repeated implementation of AR leads to domain and model shifts.

We argue that these shifts should be considered as an expected external cost of individual recourse and call for a paradigm shift from individual to collective recourse in these types of situations. We demonstrate empirically that these types of dynamics do in fact occur and introduce mitigation strategies.

Probabilistic Machine Learning — Uncertainty Quantification

Effortless Bayesian Deep Learning through Laplace Redux

Stable Dev Build Status Coverage

LaplaceRedux.jl (formerly BayesLaplace.jl) is a small package that can be used for effortless Bayesian Deep Learning and Logistic Regression trough Laplace Approximation. It is inspired by this Python library and its companion paper.

Plugin Approximation (left) and Laplace Posterior (right) for simple artificial neural network.

Simulation of changing posteriour predictive distribution. Image by author.

ConformalPrediction.jl

Stable Dev Build Status Coverage Code Style: Blue ColPrac: Contributor’s Guide on Collaborative Practices for Community Packages Twitter Badge

ConformalPrediction.jl is a package for Uncertainty Quantification (UQ) through Conformal Prediction (CP) in Julia. It is designed to work with supervised models trained in MLJ (Blaom et al. 2020). Conformal Prediction is distribution-free, easy-to-understand, easy-to-use and model-agnostic.

Conformal Prediction in action: Prediction sets for two different samples and changing coverage rates. As coverage grows, so does the size of the prediction sets.

More Resources 📚

Read on …

  • Blog post introducing CE: [TDS], [blog].
  • Blog post on Laplace Redux: [TDS], [blog].
  • Blog post on Conformal Prediction: [TDS], [blog].

… or get involved! 🤗

References

Blaom, Anthony D., Franz Kiraly, Thibaut Lienart, Yiannis Simillides, Diego Arenas, and Sebastian J. Vollmer. 2020. “MLJ: A Julia Package for Composable Machine Learning.” Journal of Open Source Software 5 (55): 2704. https://doi.org/10.21105/joss.02704.
Goodfellow, Ian, Jonathon Shlens, and Christian Szegedy. 2015. “Explaining and Harnessing Adversarial Examples.” https://arxiv.org/abs/1412.6572.
Upadhyay, Sohini, Shalmali Joshi, and Himabindu Lakkaraju. 2021. “Towards Robust and Reliable Algorithmic Recourse.” Advances in Neural Information Processing Systems 34: 16926–37.